Transparent optical module using pixel patches and associated lenslets

ABSTRACT

A transparent optical module system or device comprising an optical architecture hierarchy based on a patch unit. In aspects, the transparent optical module comprises a display sparsely populated with pixels. The patch unit comprises one or more regions of display pixels, or a pixel pattern(s), and an associated lenslet, for example on a microlens array. The lenslet is capable of transmitting display-emitted light to an eye of the wearer of the transparent optical module, which then focuses the light to form a retinal image, which is seen or perceived by the wearer. The patch units can be combined further into patch groups, wherein members of a group serve a similar role in retinal image production as a patch unit and/or lenslet. This hierarchy allows the system to be scaled to larger and more complex systems.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of and relies on thedisclosures of and claims priority to and the benefit of the filingdates of U.S. patent application Ser. No. 17/103,703 filed Nov. 24,2020, which claims priority to U.S. patent application Ser. No.16/859,092 filed Apr. 27, 2020, which claims priority to U.S. patentapplication Ser. No. 16/449,395 filed Jun. 22, 2019, which claimspriority to U.S. patent application Ser. No. 16/289,623 filed Feb. 28,2019, which claims priority to U.S. patent application Ser. No.16/008,707 filed Jun. 14, 2018, which claims priority to U.S.application Ser. No. 15/994,595 filed May 31, 2018, as well as severalU.S. Provisional patent applications. See Application Data Sheet fordetails.

The disclosures of each of the-above referenced applications and thosein the Application Data Sheet are incorporated by reference herein intheir entireties.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention is directed to operation of a near eye augmentedreality (“AR”) or mixed reality (“MR”) system that integrates thevirtual image generated by the near eye display to the real environmenttaking account of the visual processes of the retina, the visual cortexand the oculomotor system. Specifically, the current invention isdirected to transparent optical module (“TOM”) architecture, which isbuilt and/or configured in a manner to provide system scalability.

Description of Related Art

Currently existing AR or MR systems in most cases have a large formfactor and are awkward to use, heavy, demanding of high power, and/orexpensive. For these systems to have an increased level of adoption, atransformational technology change or innovation is needed to improvewhat is currently available. In addition, it is important that any suchinnovation can be easily adapted to current, established eyewear andophthalmic lens manufacturing and distribution. The innovation disclosedherein teaches such a transformational breakthrough for the AR and MRindustries. In this regard, the current innovation can also be used withvirtual reality and enhanced reality.

SUMMARY OF THE INVENTION

In embodiments of the present invention, a transparent optical module(“TOM”) system or device comprises an optical architecture hierarchybased on a patch unit. In aspects, the module may be transparent,transmissive, partially transparent, partially transmissive, opaque,partially opaque, or combinations thereof. In aspects, the patch unitcomprises one or more regions of display pixels, or a pattern(s) orpixels, and an associated lenslet, for example on a microlens array(“MLA”). The lenslet is capable of transmitting display-emitted light toan eye of the wearer of the TOM, which then focuses the light to form aretinal image, which is seen or perceived by the wearer. The patch unitscan be combined further into patch groups, wherein members of a groupserve a similar role in retinal image production as a patch unit and/orlenslet. This hierarchy allows the system to be scaled to larger andmore complex systems.

Accordingly, TOM architecture is built in a manner that provides systemscalability. In general, there are two basic elements of TOMarchitecture as described herein: a patch unit (optical) and a patchgroup (layout). A lenslet focuses a display pixel or patch to a retinalspot or portion. The lenslet field of view supports imaging an area ofan xLED display, such as a pixel patch, and in aspects, “xLED” may beused to refer to, cover, or describe, but is not limited to: OLED(organic Light Emitting Diode), TOLED (transparent OLED), microLED (alsoμLED and mLED), iLED (refers to microLED, inorganic LED), PMOLED andAMOLED (Passive Matrix and Active Matrix OLEDs), QD-LED (quantum dotLED), or combinations thereof.

A lenslet and its pixel patch will generally be referred to herein as apatch unit. In aspects, multiple patch units are used to build a largermosaic image on the retina. Due to magnification, a sparse set ofpatches and lenslets (i.e., patch unit(s)) can produce a full image asperceived by a user; accordingly, patch units are capable of beingsparsely distributed due to magnification. The array of lenslets isagain generally referred to as a MicroLenslet Array or Microlens array(“MLA”). In aspects, the display pixel patches form a sparsemicrodisplay. The intermediate area between primary patch units allowsinclusion of additional patch units to provide extra functionality. Setsof patch units that produce retinal images that overlay each other (orin cases are separate or projected side-by-side on the retina) aresometimes defined herein as a patch group. In aspects, patch units areconfigured so that lenslets do not overlap. In cases, it may benecessary to shield light to control stray light transmitted in betweenlenslets or potentially couple to neighboring lenslets. Since patchunits in a patch group can act individually, in aspects, a patch unit orunits can be illuminated independently to support different functions. Agrid of patch units can be rectangular or hexagonally packed or anyother shape. Rectangular packing, in cases, simplifies some aspects ofdetermining how to decompose the digital image data into sub-images. Thenotion of the patch unit and patch group or group of pixel patchesapplies to both static and dynamic MLAs. In aspects, a dynamic (oractive) MLA may refer to, cover, or describe one or more of:

switchable MLA;tunable MLA;electrically switching lenslets;nematic phase lenslets;smectic phase lenslets;liquid crystals;cholesteric liquid crystals;polymer encapsulated liquid crystals;nano-scale polymer encapsulated liquid crystals;blue phase liquid crystals;electrowetting lenslets;electrostatic lenslets;ferrofuidic lenslets;dielectrophoretic lenslets;pressure actuated liquid lenslets;micro-mechanical variable focus lenslets;elastomeric membrane lenslets;mechanically stretchable lenslets;chemically ion activated lenslets; and/oracousto-optical lenslets.

In embodiments, the invention is a system for producing an imageincluding a see-through near eye optical module comprising a see-throughnear eye display and a micro-lenslet array, wherein the see-through neareye display comprises a plurality of light-emitting pixels or pixelpatterns and the micro-lenslet array comprises one or more lenslets;wherein the plurality of light-emitting pixels are configured as a pixelpatch; wherein a pixel patch in optical communication with the one ormore lenslets is configured as a patch unit; and wherein the see-throughnear eye display and the micro-lenslet array are capable of permittinglight rays from a physical object in an external environment to passthrough the see-through near eye optical module to the user's retina.The light from the display and the real world external environmenttogether are capable of generating augmented reality, mixed reality,enhanced reality, virtual reality, etc. In aspects, the TOM and/oroptical module is hermetically sealed.

For embodiments, there are various optical terms and parameters thatdescribe the operation and performance of the lenslet and the patch unitas described herein. These terms include, but are not limited to,magnification, field of view, resolution, and visual acuity, by way ofexample. Some of these parameters may influence the optical design andmanufacture of the lenslet as described herein, or as would beunderstood by one of skill in the art.

The term field of view (“FOV”) in aspects describes the (maximum)perceived angular extent of a patch unit image on the retina. This FOVcan be described as an angle or as the pixel area or pixel patch on thepatch unit. In other aspects, FOV may refer to an angle or area of apatch group. It is typically, but not always, desirable to produce alarge patch unit FOV, or the largest patch unit FOV feasible. However,factors such as the lenslet performance and stray light generation havean influence on the design FOV. For example, an aspherical lens surfaceform may be necessary to improve imaging quality for pixels near an edgeor extreme edge of the patch unit FOV. Additionally, the lenslet pitchcan be designed to vary from the center to the edge of the lenslet arrayto, in cases, improve imaging properties (e.g., by minimizing Coma andother aberrations) from the periphery of the pixel patch. The patch unitFOV according to preferred embodiments described herein is designed toproduce an image that can fill the high-resolution foveal region of auser's retina. The FOV in aspects can range from a 10-degree fullangular field up to larger values. Therefore, multiple patch groups canbe used to produce a complete wide-angle view.

In embodiments described herein, magnification may describe therelationship between the angular extent at the retina versus that of thedisplay pixel patch or patch unit. In a possible embodiment of a TOMdesign described herein, an image magnification is preferably about 7×to 10×, whereas a single lens, single display VR system would have amagnification closer to 1.

The visual acuity of the eye refers to the clarity of vision and abilityto recognize detail as described herein or as would be understood by oneof skill in the art. The typical eye is able to detect about 1 arcminute angular resolution and is sometimes presented in the form of ameasurement, such as 20/20 vision. The visual acuity depends on theretinal resolving elements (rods and cones), aberrations from the eye'slens, and diffractive effects due to aperture size. The objective of theTOM as described herein is to present information that is a suitable orpreferable match to the eye's visual acuity. For example, text should besufficiently sharp and sufficiently large to be readable, while imagesshould provide preferably well-resolved features. The application of theTOM and system described herein will in aspects determine the visualacuity level that the display, system, or TOM be able to achieve.

In aspects and as understood by one of skill in the art, stray light islight that does not follow the correct or preferred path through the TOMand may produce a glare or background haze that reduces overall contrastof the desired images. The desired image light, in aspects, includes thereal world view formed from light from the external world that passesthrough the transparent areas of the display substrate and the regionsbetween lenslets on the MLA, especially in the case of a static MLA.

The stray light for a dynamic MLA differs from a static MLA. Inembodiments, a dynamic MLA is expected to be “on” for only a fraction oftime as a wearer uses the TOM described herein. During this switched ontime, the MLA operates as a set of lenslets while the display issynchronized to emit light. During the switched off state, in aspects,the MLA behaves as though it is a transparent window, and the displaydoes not emit light. During the on state, the stray light situation willbe the same or similar as described for a static MLA. However, when thedynamic MLA is switched off, typically less stray light will be presentcompared to a static MLA.

The desired virtual image is formed from light emitted by a displaypixel or pixel patch, for example, and then directed through its patchunit lenslet to the eye forming the retinal image. Real world light thatpasses through a lenslet and is redirected is one form of stray light.Another form of stray light is display light that passes through theregion between lenslets and is therefore not focused. Also, light from apatch unit pixel or pixel patch that passes through an adjacent ordistant patch unit lenslet (and, in cases, misses its own patch unitlenslet) will be focused incorrectly and is considered stray light.

In embodiments, a pixel's or pixel patch's emitted light may be directedthrough a refractive, reflective, and/or diffractive TOM lens orlenslet. In embodiments, the lenslet may be fabricated as a curvedsurface on an optically transparent substrate. The surface may have aspherical, aspherical, or other arbitrary mathematical description. Thelenslet could also provide focusing capability by dynamically adjustingits power generating properties such as index of refraction and/orcurvature structures. The lenslet may be composed of multiple layers ofmaterial or a single material. A diffractive or holographic element maydirect the light using wave interference methods. The lenslet maycomprise a single surface on a supporting substrate, or be composed ofmultiple surfaces on dual sides of the substrate or multiple substrates.The apertures of a multiple component lenslet may be oriented in linewith each other, or not. The lenslet or lenslet set may be decomposedinto multiple areas with intervening transparent or shielding areas. Thelenslet apertures may be circular, square, hexagonal, or any arbitraryshape for optimum image quality while minimizing stray light.

The use of Liquid Crystals (LCs) is one way to realize a dynamic MLA byelectrically switching the LC's index of refraction. A number ofdifferent LC technologies may be used separately or in combinations,such as conventional nematic or smectic phase and cholesteric liquidcrystals. Additionally, polymer encapsulated LCs (PDLCs) as well astheir nano-scale variety (nPDLCs) offer, in cases, advantages forconstruction of dynamic MLAs, as they are polarization invariant andtherefore can utilize unpolarized light from conventional displays suchas OLEDs and iLEDS. Further, “blue” phase LCs also possess polarizationindependent properties.

The above LC variations can be employed to construct conventional e.g.refractive lenses as have been described in this disclosure, as well asdiffractive and holographic active MLAs.

In addition to LCs, other technologies can be employed for fabricationof active or dynamic MLAs; these include but are not limited to,electrowetting, electrostatic, ferrofuidic and dielectrophoretic andpressure actuated liquid lenses. Various micro-mechanical variable focuslenses may also be used such as elastomeric membranes, which, inaspects, are mechanically stretched. Chemically ion activated lenses andlenses that utilize the various acousto-optical effects can also beemployed in dynamic MLAs.

All the above active/dynamic MLA technologies can be used separately,and/or in combination to enhance, improve, or optimize image quality andalso to minimize undesired stray light.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate certain aspects of some of theembodiments of the present invention, and should not be used to limit ordefine the invention. Together with the written description the drawingsserve to explain certain principles of the invention.

FIG. 1 is a schematic diagram showing a possible embodiment of thecurrent invention showing a configuration of display, to lenslet(s), topupil, to retina.

FIG. 2 is a schematic diagram showing a possible embodiment of thecurrent invention showing a configuration of display, to lenslet(s), topupil, to retina.

FIG. 3 is a schematic diagram showing a possible embodiment of thecurrent invention showing a configuration of display, to lenslet(s), topupil, to retina.

FIG. 4 is a schematic diagram showing a possible embodiment of thecurrent invention showing a configuration of display, to lenslet(s), topupil, to retina.

FIG. 5 is a schematic diagram showing a possible embodiment of thecurrent invention showing a configuration of display, to lenslet(s), topupil, to retina.

FIG. 6 is a schematic diagram showing a possible embodiment of thecurrent invention showing a configuration of display, to lenslet(s), topupil, to retina.

FIG. 7 is a schematic diagram showing patch units according to currentinvention.

FIG. 8 is a schematic diagram showing possible embodiments of thecurrent invention showing configurations of display, to lenslet(s), topupil, to retina.

FIG. 9 is a schematic diagram showing a possible embodiment of thecurrent invention showing a configuration of display, to lenslet(s), topupil, to retina.

FIG. 10 is a schematic diagram showing a possible embodiment of thecurrent invention showing a configuration of display, to lenslet(s), topupil, to retina.

FIG. 11 is a schematic diagram showing a possible embodiment of thecurrent invention showing a configuration of display, to lenslet(s), topupil, to retina.

FIG. 12 is an illustration showing a possible embodiment according tocurrent invention wherein a wearer of the device/system sees virtualimages in multiple focal planes.

FIG. 13 is a schematic diagram showing a possible embodiment of thecurrent invention showing a patch group with a lenslet having fourdiffering focal lengths producing differing focal planes.

FIG. 14 is a schematic diagram showing a possible embodiment of thecurrent invention showing a configuration of display, to lenslet(s), toeye.

FIG. 15 is a schematic diagram showing a possible embodiment of thecurrent invention showing lenslets of various sizes composing patchunits.

FIG. 16 is a schematic diagram showing a possible embodiment of thecurrent invention illustrating possibility of color production viamonochrome xLED areas.

FIG. 17 is a schematic diagram showing a possible dithering embodimentof the current invention.

FIG. 18 is a schematic diagram showing a possible architectural layoutembodiment of the current invention.

FIG. 19 is a schematic diagram showing a possible architectural layoutembodiment of the current invention.

FIG. 20 is a schematic diagram showing a possible brightness enhancingembodiment of the current invention.

FIG. 21 is a schematic diagram showing a possible embodiment of thecurrent invention, including showing components for forming a virtualimage(s) as described herein and interaction with the retina of avirtual image(s) and a real world image(s), thereby creating, forexample, augmented reality.

FIG. 22 is a flowchart showing a possible embodiment of the currentinvention.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS OF THE INVENTION

Reference will now be made in detail to various exemplary embodiments ofthe invention. It is to be understood that the following discussion ofexemplary embodiments is not intended as a limitation on the invention.Rather, the following discussion is provided to give the reader a moredetailed understanding of certain aspects and features of the invention.

FIG. 1 shows one possible embodiment of an optical design that relaysdevice pixel light to a retinal spot. In the particular example of FIG.1 , the schematic shows how light from a single pixel or element 1001located on or near the display device 1002 1002 is collected by thelenslet 1003, transmitted to a wearer's eye pupil 1004, and then imagedby the wearer's eye to a spot or portion on the wearer's retina 1005. Inaspects, the lenslet aperture is preferably smaller than an eye pupilaperture, and a lenslet aperture diameter determines a resolvableretinal spot size. (The drawing is not necessarily to scale, and, inaspects, the spacing between MLA and eye will be larger than shown.)

In embodiments, the lenslet diameter will be designed or made to be lessthan the eye pupil diameter. The lenslet diameter is a factor indetermining the size of the resolvable retinal spot that can beachieved. Lenslet diameter may also influence the amount of lightcollected from the display element (e.g., a pixel, pixel patch, or patchgroup) and thus the perceived brightness of, for example, a virtualimage, as well as power efficiency of the unit, device, or system.

FIG. 2 and FIG. 3 (not drawn to scale) show the lenslet imaging a patchfield (either pixels or a pattern of pixels) on the display or TOM toform a retinal image. The optics in these examples are operating at, inaspects, infinite conjugate so that the image(s) appear to be located atinfinity. FIG. 2 , in particular, shows the on-axis and chiefrays/beams. And, in FIG. 2 , lenslet 2003 optical power, focal length,or effective focal length sets the system magnification. So, forexample, a pixel patch area 2006 appears magnified to the retina and theresult is a sparse display. In aspects, by way of example, the TOM iscapable of providing a magnification of around or about 7-10×.

FIG. 3 , in particular, shows both chief and marginal rays/beams,wherein the lenslet defines the system aperture stop. The lensletdiameter will have an influence on the field size of the pixel patchthat can be imaged. Therefore, the lenslet diameter is one of the tradevariables in optimizing the optical design; more specifically, the opticdesign is determined by several parameters that may be varied dependingon the importance of brightness, retinal resolution, TOM thickness, realworld transparency, and so forth. These parameters include lensletsurface shape, lenslet diameter and focal length, and lenslet spacing.These are typically trades that are prioritized during the designoptimization process. FIG. 3 also shows that the light spreads withincreasing distance from the lenslet. (FIG. 3 includes a ray/beam traceshowing greater detail of light emitted by pixel patch and transmittedto an eye.) In preferred embodiments, a separation distance existsbetween the eye and the TOM, referred to in some cases as eye relief, ina practical system. Therefore, vignetting and eyebox size will becomeconsiderations.

FIG. 4 shows multiple patch units (single patch unit 4007) in a patchgroup 4008 illuminating an area larger than the eye pupil. In aspects,when pixel light is focused by multiple lenslets and directed along acommon angular trajectory, the pixels will be focused to the sameretinal spot or portion. In this way, several patch images can overlayeach other at the retina. This occurs in the TOM system because thelenslet aperture size is typically smaller than the eye pupil size,although in aspects it can be the same size as the eye pupil or larger.Patch units that produce this sub-image overlay are said to be part ofthe same patch group. The overlaid images enhance the perceivedbrightness to the wearer of, for example, a virtual image.

A human or animal eye pupil changes size depending on environmentallight. For example, the human pupil size may be 2 to 4 mm in diameter inbright situations and 4 to 8 mm in darkness. In aspects, the currentinvention, such as the TOM, optical module, display, display elements,MLA, and/or MLA elements, are designed for bright or dark illuminationcircumstances. Further, in aspects, the TOM, optical module, display,display elements, MLA, or MLA elements, may be curved or have anarbitrary surface to match that of an optic, such as for eyeglasses, orfor an eyeglass lens, or for a contact lens.

In embodiments, this results in a larger eyebox size, because, inaspects, these patch units produce the same retinal image. In aspects, aratio of lenslet optical power to eye lens optical power determines themagnification of the display patch on the retina. As the magnificationincreases, ever smaller pixel sizes will be needed to produce ahigh-resolution image that avoids screen door effect. Regarding the“screen door effect,” the pixel set typically includes non-emittingareas between pixels that provide electrical connectivity.

The eyebox is a region/volume where the eye can be positioned and isable to view the image or part of a larger image (e.g., subimage)relayed to the eye from the display or part of the display (e.g., patchunit), or a distant object by the optical system over the entire Fieldof View (FOV) for a typical pupil size and for the wide range of IPD(Inter Pupillary Distance). This region can include a depth toward andaway from the system, and lateral movement or offset from side to sideor above and below a specific position; in cases a preferred or idealposition. The movement may be due to head movement, eye movement, orboth. A relatively large eyebox is desired so that the system conformsto the variation of eye position within the population when the systemis mounted to the head/face of a user.

The term patch unit is sometimes used herein to define a pixel patch andthe associated lenslet that are able to produce a retinal image. Inembodiments, the display pixel patch 5006 and the lenslet 5003 exist onseparate physical components; for example, an xLED display 5002 and anMLA 5009 as shown in FIG. 5 , although it is possible to include thepixel patch and the lenslet on the same component. In embodiments, a TOMoptical system will be composed of multiple patch units 5007 capable ofoperating independently of one another.

FIG. 5 illustrates how light from multiple lenslets/patch units are ableto simultaneously pass through the eye pupil and share in producing, forexample, a common image or parts of a common image on the retina. Inthis example, the optical axis of the patch unit is normal to the planardisplay and the lenslets have been set to collimate pixel light.Multiplicity is achieved because the lenslets 5003 have a diametersmaller than that of the eye pupil. The number of lenslets that are ableto fill the eye pupil will depend on the relative size of eye pupil andlenslet, and the trajectory that the patch unit directs light.Accordingly, FIG. 5 shows multiple patch 5007 units forming a largerretinal image mosaic 5010, and it shows that it is also feasible todirect light from multiple patch units to form a mosaic at the retinaproducing a larger image. In aspects, each pixel patch or patch unitcould contain a different subimage defined, for example, by external orinternal processors. Further, in embodiments, the xLED display and theMLA would conform to a curved surface.

FIG. 6 shows patch units 6007 arranged on curved surfaces for both thedisplay and the MLA. This configuration may be capable of leading toimproved optical on-axis usage of the lenslets and the imagingperformance. In this situation, the TOM will also conform to the shapeof eyeglass optics to which it can be attached, connected, embedded, orassociated.

1.1 Patch Group Architecture

The optical architecture, in aspects, comprises a lenslet imaging adisplay pixel or patch to a spot or portion on the retina. In aspects,the lenslet is able to image an extended field of view on the displaycomposed of a patch of pixels that forms a sub-image on the retina. Thedisplay patch unit can be replicated in an array fashion using a sparse,mostly sparse, or partially sparse micro-display device and amicro-lenslet array to produce a larger composite retinal image. Inaspects, the display pattern projected by a primary patch unit willdiffer from the others. Due to the short focal length of the lenslets inthis example of the system, the display magnification is relativelylarge, ranging from about 5× to 10×, by way of example. Therefore thelateral distance between critical primary patch-lenslet sets (e.g.,patch units) will be larger than the lenslet diameter.

FIG. 7 shows the top view of lenslets of a patch unit layout, whereinprimary patch units 7007 are, in this example, the minimal necessary toproduce the composite retinal image. The hollow circles representadditional lenslets 7011 that can be used for patch groups. This figure,as well as others shows how to create multiple range virtual imageplanes, virtual image focal planes, virtual distances, virtual depths,or combination thereof. Overlaid retinal sub-images are produced by thepatch units, which provides advantages compared to currently existingdisplay concepts. Due to the limited field of view of the lenslets, inexamples, several patch units are needed to produce a larger mosaicimage. However, the minimal set of patch units, in this example, issparse. This is shown by the solid blue lenslet circles 7007 in FIG. 7 .The intermediate area can be filled by extra patch units that provideadditional functionality. These additional patch units are shown ashollow circles 7011 in FIG. 7 . The additional units will create retinalsub-images that overlay each other, sometimes referred to as a patchgroup. If the optical power or effective focal length of additionallenslets are designed to vary slightly from each other, then respectivesub-images appear to focus at different virtual image focal planes,virtual distances, or virtual depths, than other units. The actual patchunit size and patch group layout will depend on specified opticalparameters, such as lenslet size, pixels per patch, eyebox size, usecases of the product with integrated TOM, etc.

More specifically in FIG. 7 , for this example, solid color circles 7007represent the minimal set of primary patch units to create the compositeretinal image and hollow circles 7011 are additional patch unitsinserted between primaries. In the case shown, the additional patchunits produce multiple virtual focal image planes, distances, depths, orcombinations thereof. Rather than leaving the space between patchesempty, it is possible according to the current invention to insert morepatch units in order to increase system functionality. These additionalpatch units (shown as hollow circles in FIG. 7 ) that project a commonpattern to form like sub-images are sometimes referred to herein as“patch groups.” Patch groups are able to increase the image brightness,increase the eye box size, vary in size as needed for resolution, ditherthe sub-images to smooth the image, and provide multiple virtual imageplane focal distances. Assigning patch units into the space betweenprimary patch units will reduce the sparseness of both the display andMLA and will thereby lead to reduced transparency for real world viewlight (see, e.g., FIG. 21 ). In embodiments, each patch group producesredundant images, therefore individual patch units comprising the groupcan be dispersed in locations over the display or have slightly varyingfeatures to provide unique functionality for the TOM. These optionsinclude:

-   -   Increased eyebox coverage (described in more detail in section        1.2.1);    -   Gaze dependent views (described in more detail in section        1.2.2);    -   Increased image brightness (described in more detail in section        1.2.3);    -   Enabling multiple virtual image focal planes, distances, depths,        or combinations thereof (described in more detail in section        1.2.4);    -   Variation of aperture size supporting direct and peripheral gaze        angles (described in more detail in section 1.2.5);    -   Color image production from monochrome patches (described in        more detail in section 1.2.6); and    -   Patch unit dithering to smooth screen door effect (described in        more detail in section 1.2.7).

1.2.1 Patch Groups Used to Create a Larger Eye Box

The eyebox size is a system consideration since mounting an AR, MR, orvirtual reality (“VR”) unit should account for the variation ofinter-pupil distance and eye position variation of the user population.Also, the eye pupil moves as the gaze shifts to an alternate angle. Theeyebox is the volume that the eye can be located within and still viewthe full display. Eyeball rotation can move the pupil a few millimetersfrom its straightforward gaze position.

Since the eyebox for an individual patch unit is, in aspects, about thesize of the lenslet diameter, a minor shift in the gaze angle wouldnormally lead to partial or complete vignetting of the patch unit light.However, multiple patch units can be designed according to the currentinvention to produce similar overlapping retinal images no matter howthey are arranged across the display device. Thus, the eyebox size canbe increased by including multiple patch units in the architecture.

FIG. 8 shows light rays collimated into beams by a lenslet 8003 from thecenter and extreme pixels on a pixel patch. The arrows at the right 8012of FIG. 8 show how the extent of the rays/beams diverge with distancefrom the lenslet. Eventually the divergence can become so large that theeye pupil may not capture all rays/beams leading to vignetting of thepixel patch. Thus, FIG. 8 effectively illustrates propagation of lightrays emitted by patch pixels and focused into beams by a lenslet. Thisconfiguration focuses the light to appear to the eye as though the pixelpatch is located at an infinite distance (infinite conjugate). The raysfrom each pixel in this case form constant diameter beams traveling awayfrom the lenslet. As the beams propagate, they diverge from the centeron-axis beam. The vertical arrows at the extreme right 8012 in FIG. 8show how beams separate at a distance from the lenslet. If the lensletwere positioned just outside the eye pupil, and if the lenslet diameterwere smaller than that of the pupil, then all the light would becaptured and imaged by the eye. Such would be the case if the patch unitcould be mounted within or near the location of a contact lens. However,in some cases, an eye relief distance will be needed so that the TOMmodule is clear of the region in the vicinity of the eye similar to theeye relief provided by eyeglasses. The TOM according to the presentinvention is designed to, in embodiments, be attached to, connected to,embedded in, or associated with an eyeglass optic and will thus be, incases, about 15 to 25 mm from the front of the eye. Therefore the beamsmay diverge sufficiently that the pupil obscures a portion of the light.When this happens, the patch image appears vignetted, or cut off alongan edge. Therefore eye relief will be a consideration in designing thelayout of the patch groups to avoid, for example, vignetting.

FIG. 9 shows a multitude of patch units 9007 used to illuminate a largeregion at the eye's distance, which is capable of creating a largereyebox region. Only the central pixel rays and beams are shown here.FIG. 10 illustrates that when the eye's gaze shifts upward in thedrawing (as compared to FIG. 9 ), the eye pupil is still able to viewpatch units that create, in cases, identical images. The patch imageshifts away from the center of the retina's foveal region since it is nolonger expected to be the straight-ahead view of the gaze (as in FIG. 9). The patch units do not have to be packed densely across the eyebox;however, there should be an adequate number being viewed simultaneouslysince the changing gaze angle will lead to patch unit vignetting and achanging number of overlapping images as the gaze shifts. Accordingly,as illustrated in FIGS. 9 and 10 , shifting the eye gaze angle allows acommon image to be viewed over a larger eyebox with similar patch units;therefore, the retinal image can differ depending on where the eyegazes, and, for example, if the eye is looking up or down, it can viewan image that differs from a view looking in a different direction, andthis concept can be extended to more views as the eye changes gazedirection. This change could lead to an image brightness variation withgaze if the number of patch units viewed is, for example, less thanabout 10. Note that a flat display configuration is shown. In this casethe display patches are typically on the optic axis of the lensletleading to optimal performance. When a curved display is used, the sameeye box concept can still be used although the layout and lensletprescription may be slightly different.

1.2.2 Gaze Dependent Views

Reviewing the patch group or patch unit layouts in FIG. 9 and FIG. 10 ,it can be understood how it is possible according to the presentinvention to build a gaze dependent image. That is, when the wearer'seye looks up or down (or left and right), it is possible for the TOMdescribed herein to provide distinctly different views or images. Thisis shown in FIG. 11 , for example. Here, the red 1113, green 1114, andblue 1115 arrows indicate that pixel patches, patch groups, and/or patchunits are transmitting different images that the eye's pupil willintercept in the eyebox. The figure shows only the collimated light fromthe center patch pixel. The actual patch cone of rays/beams for a patchunit will be spread over a larger area, in aspects. Therefore the gazedependent views will change and blend in the transition region betweenpatch units. An application might use this gaze dependent technique toprovide an alert in the straight ahead view that directs the user toglance to the periphery to obtain display data that might otherwiseobscure the critical direct ahead view. Accordingly, varying the patternacross a patch group can lead to gaze dependent viewing, and here, asshown in FIG. 11 , the upper and lower gaze images (or, e.g., left andright and/or diagonal) will differ from that of the center gaze. Sincethe pixel patch and/or patch unit image varies according to location,the retinal image differs depending on where the eye gazes.

1.2.3 Promoting Increased Brightness

When the retinal images from two or more patch units are aligned, theperceived image brightness increases. This is a consideration when thexLED display provides limited illumination power and the TOM system isused in a bright outdoor environment. The patch units can beconcentrated in close proximity to each other on the display to promotethe increased brightness.

FIG. 20 shows a set of patch units that similarly direct light throughthe pupil to produce overlaid images that increase the apparentbrightness of the retinal image. The figure shows multiple lenslets 2003in an MLA directing light equivalently over an area larger than that ofthe eye pupil. This configuration increases the size of the eyebox, too.For increased brightness, the patch units of a patch group may beconcentrated together. For a greater eyebox size, the patch units can bedistributed more sparsely.

1.2.4 Producing Multiple Focal Planes

In some cases, the system uses a fixed focal length for all MLA lensletsand a fixed display to MLA separation to produce an image the eyeinterprets as being located at a common focal distance from the user.This common virtual focal plane might appear to be located at infinityor at a closer location such as 6 feet distance depending on the lensoptical design.

However, it would be further desirable to adjust the display virtualimage to appear at a focal distance consistent with real world objectsin the viewer's scene of view. In addition, for binocular systems, thedifference between focus accommodation and eye convergence can lead tofatigue or more severe eye strain symptoms during extended wear if notcorrected. This multitude of virtual image focal planes, distances,depths, or combinations thereof, is illustrated in FIG. 12 , wherein awearer's perception of multiple virtual image focal planes, distances,depths, or combinations thereof, may be produced via TOM patch groupsand/or patch units. FIG. 12 shows, in part, individually illuminatedvirtual image planes as viewed by a user. In aspects, softwareapplications will generate the display image for each virtual imagefocal plane, distance, or depth, and the mechanism is described ingreater detail in the explanation of FIG. 13 and FIG. 14 herein. Themulti-virtual image focal plane range may have a greater impact forbinocular systems where eye convergence and accommodation, if notmatched, can lead to viewer fatigue.

The apparent virtual image focal plane could be adjusted if thedisplay-to-lenslet separation could be mechanically varied; however, incases, this would introduce undesirable complexity into the system andan avenue for potential system failures. The TOM system according to thepresent invention is able to address this need for multiple virtualimage focal planes by varying virtual display focal power or effectivefocal length among the lenslets, and in cases the lenslets within apatch group.

Generally, two or more patch units of a patch group are simultaneouslyable to project light through the eye pupil. As an example, FIG. 13shows a patch group composed of a 2×2 array of patch unit lenslets. Thenumbers on the Figure lenslets identify the focal plane distances of 1,2, 3, and 4+ meters. The most distant focal plane will be designed tothe eye's hyperfocal distance, thereby accommodating an infinitedistance plane, for example. The number of focal planes can bedetermined by application specifications and limited by the number ofpatch units that fill the space available for a patch group. The valueof the optical power or effective focal length between the focal lengthsmay vary by less than 1%, so the lenslets are virtually identical, inaspects. A custom MLA, in cases, may need to be designed and fabricatedwith the lens form varying across the component, which would beunderstood by one of ordinary skill in the art.

More specifically, in FIG. 13 , the illustration shows an example of apatch group lenslet set inside an MLA composed of an array of 2×2lenslets 1304. Each lenslet has a slightly different optical power oreffective focal length. The lenslet labels indicate optical designs thatproduce a 1 meter, 2 meter, 3 meter, and infinite distance or depthvirtual image focal plane. The number of lenslets in the patch grouplenslet set can be larger than 2×2. The overall size of the patch groupshould preferably not be larger than the dimensions of the eye pupilsize. For example, if the eye pupil diameter is 4 mm, and a lensletpitch is 600 microns, the number of embedded lenslets is √2*4/(0.6)=9.4,which supports a 9×9 lenslet patch group set. However, lenslets can alsoserve functions other than producing virtual image planes, since thismany display planes may be larger than needed.

The ability to present a virtual image focal plane, distance, or depthis illustrated in FIG. 14 , which shows light ray/beam traces of twopatch unit 1407 configurations. At the top of FIG. 14 , the lenslet andpixel patch (patch unit 1407) are set to operate at infinite conjugate.The virtual image plane appears to be at infinity. (Infinite conjugateconfiguration—Blue light rays from display spot are focused to acollimated beam. Lenslet patch separation, d, is equal to lenslet focallength, f, when spacing is air gap. (f=d.) Red rays show apparent eyegaze. Virtual image appears to be at infinity when viewed by eye.)

At the bottom of FIG. 14 , the lenslet optical power or effective focallength is higher, greater, or longer, so the rays/beams appear to emitfrom a closer virtual image focal plane. (Finite conjugateconfiguration—Optical or focal power is less than that of lenslet ininfinite conjugate configuration. Effective focal length is longer thaninfinite conjugate case. (f>d.) Red rays show apparent eye gaze. Virtualimage plane appears to be at a distance closer than infinity when viewedby eye. Difference in optical power or effective focal length may besmall/minor.) The two patch units 1407 may be similar except that thebottom configuration has a weaker effective focal length lenslet, sothat the rays/beams seem to diverge from a virtual image point making itappear that the imaged pixel location is at a focal plane closer thaninfinity. Thus, by varying lenslet optical powers or effective focallengths across the MLA, a variety of virtual image focal planes,distances, depths, or combinations thereof, can be produced.

The presentation of a particular virtual image focal plane, distance, ordepth to the user can be managed by system electronics. For example,when a single focal plane is desired, then only one specific pixel patchof a patch group will be powered to illuminate. If a virtual image(s) iscomposed of multiple focal planes, then multiple patches will beilluminated and the electronic processing engine will determine theappropriate pixel sets to light on each patch unit so that the retinalimage does not contain confusing overlap. The processor will alsoaccount for the variation in magnification between patch units (e.g.,due to varying optical powers or effective focal lengths of relevantlenslets). It may also, in cases, be necessary to include eye trackinghardware and software to determine the user gaze. In aspects, one ormore processors and/or accompanying software or applications willdetermine the necessary optical power or effective focal length of oneor more lenslets in order to implement the effect described abovewherein intentionally changing or choosing the optical power oreffective focal length of one or more lenslets creates a perception thatvirtual images are at varying focal distances, planes, or depths asperceived by a person wearing the TOM. For example, a processor maydetermine that a virtual image should appear at a certain distance froma user and activate patch units and/or patch groups having lensletoptical power or effective focal length suitable to make the virtualimage appear at the desired distance from the user (or, in aspects,instruct relevant lenslet(s) to change optical power or effective focallength to make the image appear at the desired distance from a user).The optical power or effective focal length to achieve the desiredperception will be set by the one or more processors and the settings orranges can be predetermined or determined by the one or more processorsdynamically depending on circumstances such as the virtual image size orshape, the real world input, movement, visual cues, brightness, thedesired distance, other real world or virtual objects, and otherfactors.

FIG. 22 shows a possible embodiment of a method one or more processorsmight take to allow the TOM to allow for creation of the perception bythe user that virtual images are at different distances from a user ortake up different space in a real and/or virtual environment. Forexample, the processor may identify the virtual image or what distance avirtual image should appear from a user whether in real or virtual spaceand depending on where the user and where the virtual image/object arein space 2201. The processor can activate a particular patch unit orpatch group having a lenslet(s) with an optical power or effective focallength preferable for creating an image at the desired perceiveddistance (or, in aspects, a lenslet can be modified to create theperception of distance) 2202. Thus, the processor can determine theoptical power or effective focal length that the lenslet should be (orchange to be) in order to make the virtual image/object appear at thedesired distance 2202. The processor will activate a chosen patchunit(s) and/or patch group(s) having a lenslet(s) with an optical poweror effective focal length to achieve the perception of the desireddistance of the virtual image/object 2203. In other aspects, theprocessor will instruct the relevant lenslet or lenslets to changeoptical power or effective focal length as decided by the processor thatis appropriate to achieve the perception of the desired distance of thevirtual image/object 2203. The processor is capable of dynamicallycommunicating internally and with elements on the TOM so that differentvirtual images that the processor determines should be at differentdistances from the user can appear to be located in space at differentor multiple virtual image focal planes, distances, or depths in space orrelative to one another 2204. Moreover, as the user moves in space orthe virtual images are moving and distances between user and virtualimages are changing, the processor is capable of recognizing thesechanges and dynamically carrying out the method so that chosen patchunits and/or patch groups having lenslets with preferred optical poweror effective focal lengths will be activated (or, in aspects, so thatlenslet(s)′ optical powers or effective focal lengths are capable ofchanging) as the user moves in space or the virtual images move. Thiswill create the perception to the user, for example, that as the usermoves towards a virtual image, the virtual image/object is gettingcloser to the user. As the user moves farther and closer to virtualimages, the processor will account for changing desired distances andactivate different patch units or patch groups (or instruct lenslets tochange) and account accordingly so that from the perspective of the userhe/she is moving around in space or objects are moving around in avirtual or real world environment 2205. Moreover, because the lenslets,groups of lenslets, patch units, and patch groups are capable ofoperating independently, one virtual image can appear at one distanceand another virtual image can appear at a different distance.Accordingly, the perceived distance of the virtual images from the user(or relative to one another) can change, and the processor can accountfor the perception of these virtual images in space as they or the usermove by changing which patch units or patch groups are activated (or bychanging lenslet(s) optical power or effective focal length) inreal-time or near-real-time, in embodiments, to create perception ofmovement and varying changes in distance and perspective, including theappearance of different virtual images at different distances from theuser.

In embodiments, the system is capable of creating the perception of athree-dimensional object in space by simultaneously activating multiplepatch units having different lenslet optical powers or effective focallengths to create a 3D virtual image. By way of example, patch unitshaving an optical power or effective focal length to make parts of theobject appear closer to the TOM will activate simultaneously asdifferent patch units having different optical powers or effective focallengths to make parts of the same object appear farther from the TOM. Inother words, several virtual image focal planes, depths, or distancescan be viewed or perceived simultaneously to produce the appearance of a3D object to the user. Therefore, in embodiments having static lenslets,multiple patch units will be on or activated to create the virtual 3Dview. Further, for the static case, this means, in aspects, multiplespatially diverse patch units are operating. For a TOM withdynamic/active lenslets according to the current invention, in aspects,the choice of differing virtual image focal planes can be made duringthe time sequence of the view. In aspects, one or more processors canpower the determination of where objects or parts of objects shouldappear in space, which lenslet optical power(s) or effective focallength(s) is suitable or preferable to create the 3D perception ofdifferent distances/locations, which patch units should be activated tocreate the perception of 3D depending in cases on the lenslet opticalpower or effective focal length of those patch units, or in cases ofdynamic/active MLAs changing optical powers or effective focal lengthsof particular lenslets. For both static and dynamic/active MLAs, theprocessor can make the determinations and activate or change theproperties in real-time or near-real-time so that the viewer perceivesthat the viewer is moving in 3D space or that objects are moving in 3Dspace.

In embodiments, once the processor or system reaches step 2205 of FIG.22 , the process repeats itself by starting back at step 2201. Inaspects, depending on the needs of the system, the processor is capableof skipping steps, repeating steps, going back to prior steps, goingahead to later steps, or otherwise acting dynamically.

1.2.5 Variation of Aperture Size to Support Direct and Peripheral GazeAngles

The lenslet diameter will influence retinal image resolution and thesize of the pixel field that can be reasonably imaged. In general, thesystem will preferably have high resolution in the eye's foveal region,which leads to large aperture lenslets to support this. However, inaspects, peripheral images can be lower resolution and thus use smallerlenslets that may need to be used off-axis. Therefore, the MLA can becomposed of multiple aperture size lenslets. In this manner, patch unitssupporting the concept of a patch group will be distributed across arelatively large region of the MLA.

For example, in particular layout embodiments, lenslets that support adirect gaze angle would have the largest apertures and would generallybe located near the center gaze angle of the MLA. Peripheral gazelenslets could have less, such as one half to one quarter the aperturediameter of the lenslets supporting direct gaze angle, and they could beinterspersed among other lenslets as shown in FIG. 15 . In the picturedregion of the MLA, there are large lenslets supporting direct gazeimaging, with interspersed small lenslets supporting peripheral gazeangles. In examples, MLA fabrication could accommodate various lensforms and sizes across the array.

More specifically, in FIG. 15 , variation of aperture size may influenceimage resolution and FOV. In embodiments, direct gaze patch units shouldhave high resolution, large aperture lenslets. In embodiments,peripheral gaze patch units can have lower resolution, smaller aperturelenslets. Generally small aperture lenslets (large F #'s) will have alarger display field, but the size of the diffraction limited spot willbe greater. Small aperture lenses produce less spot aberrationtypically. Larger diameter lenslets (smaller F #'s) will have moreaberrations and a smaller display field. They will produce highresolution spots, but may require aspheric designs. The larger lensletsalso couple more light to the retina and will produce brightersub-images.

1.2.6 Composite Color Production Using Monochrome Display Patches

Current xLED displays typically have composite pixels composed of red,green, blue, and possibly white subpixels. When the display is vieweddirectly by the eye or through a low magnification objective lens, thesubpixels blend together without noticeable visual effect. However, insome examples, a TOM presents a high magnification image of the pixelpatch, so when the sub-pixel size is sufficiently large (>5 microns) thefoveal region of the eye may resolve each sub-pixel and the black spacebetween them. This may lead in some cases to a screen door effect andthe colors may not desirably blend.

For this reason, in aspects, it may be beneficial to use monochromepixel patches to achieve desirable resolution and blend multiple patchunit images to produce the color effect. The perceived color is due toretinal image overlay from monochrome patches in a patch group. Asuitable number of patch units should be used that are simultaneouslyviewed so that the color does not shift during eye gaze movement.

FIG. 16 illustrates color production via monochrome xLED areas 1616 whenmonochrome patches are simpler or less expensive to manufacture and willstill support the TOM device and system disclosed herein. In cases, whenpixels are composed of multiple color sub-pixels, the overall resolutionmay be less. Collecting monochrome subpixels into a single patch 1616and then overlaying three or more patches can produce a color image withhigher resolution in this case. This technique may be used if, forexample, producing sets of color pixels is more difficult or expensivethan producing monochrome sets. Note that, in particular aspects, thepixels could also be white, and the lenslet could carry a color filter.This, however, in cases, may be a less efficient use of the light.

1.2.7 Dithering

The patch units of a patch group can be varied so that there is asubpixel size offset between the various overlapping retinal images.This technique can be used to smooth a screen door effect (or in casescolor banding) making the resulting image, in aspects, more pleasant toview for the user. The technique also allows the perceived image toappear to have a smoother, higher resolution appearance.

According to the current invention, the dithering technique is producedusing sets of patch units that are concentrated in location so that theysimultaneously or nearly simultaneously image onto the retina. Inaspects, pixel layout on the microdisplay may be fixed, therefore thelayout of the MLA may be chosen for customization. In other aspects, thepixel layout and/or MLA may allow for custom configurations. Thecenter-to-center pitch of the lenslet may be set so that it is not equalto an integral number of pixels, and, in aspects, the offset from theinteger multiple may be set to a fraction of a pixel. For example, ifthe pixel pitch is 10 microns, the lenslet center-to-center pitchbetween a lenslet and its three neighbors in a 2×2 dithering formationcould be about 105 microns along each dimension. In a 3×3 ditheringsituation, a 103.3 micron offset might be selected, for example.

FIG. 17 shows simulated examples of patch group dithering to reduceretinal image screen door effect. On the top of FIG. 17 , a sparselypopulated display pattern is shown 1710. In the center, a 2×2 ditheringpattern is shown 1720. On the bottom, a 3×3 dithering pattern is shown1730. In other words, FIG. 17 shows a sequence of simulated imagesproduced using a Zemax optical design and modeling computer application.The image on the top is a single patch unit output. The display pixelsare distinctly separated resulting in spacing between imaged spotssimilar to an exaggerated screen door effect. The middle image shows theimage overlay where a half pixel offset is introduced in a 2×2 set ofpatch units. The bottom-most image shows the image where a one-thirdpixel offset is used in a 3×3 set of patch units.

In general, dithering herein means producing overlaid sub-images thatare each translated by a fraction of a pixel pitch with regard to eachother. Dithering may be useful if microdisplay pixels are significantlylarger than the eye can resolve, such as resolving the spacing betweenpixels. Dithering can provide retinal image smoothing. Dithering canalso be used to produce interlaced retinal images that create higherresolution images. Assuming that the microdisplay pixel layout isregularly spaced and permanent, the dithering effect may be achieved byshifting the lenslet location in a patch group by a fraction of a pixelrelative to a reference lenslet. The shifted lenslets should preferablyoccur in both horizontal and vertical dimensions.

2.1 Patch Group Layout

In embodiments, patch units sharing a common retinal image can beassigned to patch groups. How those patch groups can be laid out acrossthe TOM display—or the layout architecture—will depend on the patchgroup function and their use in enabling a particular application. Inaddition, the chosen real world view transparency can determine thenumber and density of the patch units.

FIG. 18 shows one example of a dense patch group layout. (Concentratedpixel group layout. Colors and rectangular outlines indicate patch unitsof a common patch group (e.g., patch group 1 1810, patch group 2 1820,patch group 3 1830, patch group 4 1840, patch group 5 1850, and patchgroup 6 1860).) The patch group's patch units are identified by colorwithin a square (patch group) showing the extent of the patch group (thepatch units are notated by circles within the patch group squares (e.g.,1801)). In this example, a 3×3 set of circular lens patch units aregrouped shoulder to shoulder (or next to one another). In thisconfiguration several patch group members (patch units) will likelyilluminate the eye pupil simultaneously or nearly simultaneously. Thislayout will prove useful for color generation, brightness enhancement,and dithering options.

FIG. 19 shows a distributed and interlaced set of patch groups. In anembodiment, the patch units are identified by differently coloredcircles; circles with the same color are patch units of the same orcommon patch group in this particular embodiment. In this example, thereare two patch units from other groups separating units of a patch groupboth horizontally and vertically. In this configuration, a few membersof a patch group may illuminate the eye pupil simultaneously. Thisconfiguration would be useful for increasing eyebox size, gaze dependentviewing, or field of view.

It is also possible that a hybrid approach of a concentrated anddispersed layout will be preferable. The layout configuration will bedetermined during the design of the architecture for a specificapplication. Even though the layout has been shown configured as arectangular grid array in the Figures, a hexagonal packing or arbitrarypattern may be selected. This arbitrary nature may prove necessary oncurved surface TOMs.

Further regarding FIG. 19 , the patch unit layout in a patch group maybe determined by functionality. In this particular example/embodiment,similar color circles designate elements that make up a patch group. Inthis embodiment, pixel patches and/or patch units are interlaced, whichis a way to increase eyebox size.

Accordingly, as disclosed herein, multi-range patch groups and a patchgroup(s), including pixels, pixel patches, lenslets, and patch units,provide benefits. Patch groups can be applied to both static and dynamicMLA-based architectures. Regarding uses for patch groups, benefitsinclude but are not limited to:

Image brightness increase due to pixel patch or patch unit overlay;Increased eyebox size when patch group is sparsely distributed;Creation of multiple virtual image focal planes, depths, or distances asa result of lenslet optical power or effective focal length variations;Varied field of view image quality vs. aperture size;Color production via monochrome xLED areas; andDithering for enhancing resolution or reducing screen door effect.Note—Pixel patches generally are smaller in size than lenslets, but notalways.

The invention herein includes several Aspects, including:

Aspect 1: An optical system comprising:

a see-through near eye optical module comprising a see-through near eyedisplay and a micro-lenslet array, wherein the see-through near eyedisplay comprises one or more plurality of light-emitting pixels and themicro-lenslet array comprises one or more micro-lenslets;

wherein one of the one or more plurality of light-emitting pixels isconfigured as a pixel patch, and wherein the see-through near eyedisplay comprises a plurality of pixel patches;

wherein the pixel patch is in optical communication with a micro-lensletof the one or more micro-lenslets, wherein the pixel patch in opticalcommunication with the micro-lenslet is configured as a patch unit, andwherein the see-through near eye optical module comprises a plurality ofpatch units;

wherein a first image or subimage and a second image or subimage fromtwo or more of the plurality of patch units are directed along a commonangular trajectory towards a user wearing the see-through opticalmodule, wherein the first image or subimage is identical to the secondimage or subimage, wherein a first position of an eye of the user, ahead of the user, or both the eye and the head of the user, results inthe first image or subimage being directed to a portion of a retina ofthe user, and wherein when the user changes lateral displacement of theeye of the user, the head of the user, or both the eye and the head ofthe user, to a second position, the second image or subimage overlapssome or all of the portion of the retina of the user where the firstimage or subimage was directed; and wherein the see-through near eyedisplay and the micro-lenslet array permit light rays from a physicalobject in an external environment to pass through the see-through neareye optical module to a retina of the user.

Aspect 2: The optical system of Aspect 1, wherein the see-through neareye display is partially, sparsely, mostly, or fully populated by theplurality of the light-emitting pixels or the plurality of the pixelpatches.

Aspect 3: The optical system of Aspect 1, wherein images or subimagesfrom different patch units of the plurality of patch units are at leastone of the same, redundant, related, variant, or different, fragments ofa larger image.

Aspect 4: The optical system of Aspect 1, wherein two or more of the oneor more micro-lenslets have different optical powers or effective focallengths.

Aspect 5: The optical system of Aspect 1, wherein a first micro-lensletin a first patch unit has a different optical power or effective focallength than a second micro-lenslet in a second patch unit.

Aspect 6: The optical system of Aspect 1, wherein a first micro-lensletin a first patch unit has a different optical power or effective focallength than a second micro-lenslet in a second patch unit, and whereinthe difference in optical powers or effective focal lengths between thefirst micro-lenslet and the second micro-lenslet is capable of creatinga perception by the user that images or subimages created by the firstpatch unit and the second patch unit are at different virtual imagefocal planes, distances, depths, or combinations thereof, from an eye ofthe user.

Aspect 7: The optical system of Aspect 1, wherein a first micro-lensletin optical communication with a first group of pixel patches has adifferent optical power or effective focal length than a secondmicro-lenslet in optical communication with a second group of pixelpatches, and wherein the difference in optical powers or effective focallengths between the first micro-lenslet and the second micro-lenslet iscapable of creating a perception by the user that images or subimagescreated by the first group of pixel patches and the second group ofpixel patches are at different virtual image focal planes, distances,depths, or combinations thereof, from an eye of the user.

Aspect 8: The optical system of Aspect 1, wherein an optical power of afirst micro-lenslet of a first patch unit is less than an optical powerof a second micro-lenslet of a second patch unit or an effective focallength of the first micro-lenslet of the first patch unit is longer thanan effective focal length of the second micro-lenslet of the secondpatch unit, and wherein beams from the first micro-lenslet appear toemit from a closer virtual image or subimage focal plane, distance,depth, or combination thereof, to an eye of the user.

Aspect 9: The optical system of Aspect 1, wherein micro-lenslet opticalpower or effective focal length differences across a micro-lenslet arraycomprising the micro-lenslets are capable of creating a perception bythe user that images or subimages are appearing in more than one virtualimage focal planes, distances, depths, or combinations thereof.

Aspect 10: The optical system of Aspect 9, wherein a desired virtualimage focal plane, distance, depth, or combination thereof, a desiredpixel patch or a desired patch unit is illuminated.

Aspect 11: The optical system of Aspect 1, wherein a first micro-lensletin a first patch unit has a different aperture size than a secondmicro-lenslet in a second patch unit.

Aspect 12: The optical system of Aspect 1, wherein the optical systemincludes or supports multiple aperture size micro-lenslets.

Aspect 13: The optical system of Aspect 1, wherein one or moremicro-lenslet in a middle area of a user gaze direction have an aperturesize larger than one or more micro-lenslet in a peripheral area of theuser gaze direction.

Aspect 14: The optical system of Aspect 1, wherein one or moremicro-lenslet supporting direct gaze imaging have a larger aperture thanone or more micro-lenslet supporting peripheral gaze angles.

Aspect 15: The optical system of Aspect 1, wherein the pixel patches aremonochrome pixel patches, and wherein multiple pixel patches are blendedtogether or overlapped to produce a perception by the user of color orincreased resolution.

Aspect 16: The optical system of Aspect 1, wherein the pixel patches aremonochrome pixel patches, and wherein images from the monochrome pixelpatches are overlaid on the portion of the user's retina to create aperception by the user of color.

Aspect 17: The optical system of Aspect 1, wherein a subpixel size isoffset between a first pixel patch and a second pixel patch or a firstpatch unit and a second patch unit.

Aspect 18: The optical system of Aspect 1, wherein a center-to-centerpitch of a micro-lenslet is offset from an integral number oflight-emitting pixels.

Aspect 19: The optical system of Aspect 1, wherein a center-to-centerpitch of a micro-lenslet is offset from an integral number oflight-emitting pixels, and wherein the offset from the integral numberof light-emitting pixels is a fraction of a pixel.

Aspect 20: The optical system of Aspect 1, wherein the optical systemproduces overlaid images or subimages on the retina of the user, whereineach image or subimage is translated by a fraction of an imaged pixelpitch with respect to one another.

Aspect 21: The optical system of Aspect 1, wherein a first micro-lensletlocation is shifted by a fraction of a pixel relative to a secondmicro-lenslet.

Aspect 22: The optical system of Aspect 1, wherein a first micro-lensletlocation is shifted by a fraction of a pixel relative to a secondmicro-lenslet, and wherein the shift of the first micro-lenslet occursin both horizontal and vertical dimensions.

Aspect 23: The optical system of Aspect 1, wherein the one or moremicro-lenslets are at least one of static micro-lenslets or dynamicmicro-lenslets.

Aspect 24: The optical system of Aspect 1, wherein an optical power oreffective focal length of the one or more micro-lenslets determinesmagnification of at least one of the optical system, images, orsubimages.

Aspect 25: The optical system of Aspect 1, wherein the optical systemcomprises multiple pixel patches and/or multiple patch units capable ofoperating independently of one another.

Aspect 26: The optical system of Aspect 1, wherein the one or moremicro-lenslets have a diameter smaller than that of a pupil of an eye ofthe user.

Aspect 27: The optical system of Aspect 1, wherein the system producesmultiple virtual image or subimage focal planes, distances, depths, orcombinations thereof, as perceived by the user by overlaying retinalimages produced by patch units of the plurality of patch units.

Aspect 28: The optical system of Aspect 1, wherein a primary patch unitis supplemented by one or more additional patch units that are capableof producing an overlaying image or subimage on a retina of the user.

Aspect 29: The optical system of Aspect 1, wherein overlaid images orsubimages produced by two or more patch units increase image brightness,increase eye box size, or combinations thereof.

Aspect 30: The optical system of Aspect 1, wherein two or more pixelpatches of the plurality of pixel patches produce the same or redundantimages on a retina of the user, thereby rendering the optical systemcapable of locating individual pixel patches of the two or more pixelpatches at different areas of the see-through near eye display.

Aspect 31: The optical system of Aspect 1, wherein a first optical poweror effective focal length of a primary patch unit differs from a secondoptical power or effective focal length of a secondary patch unit,wherein the differing optical powers or effective focal lengths of theprimary patch unit and the secondary patch unit cause an image orsubimage produced by the primary patch unit to appear to be focused at adifferent virtual image or subimage focal plane, distance, depth, orcombination thereof, than an image or subimage produced by the secondarypatch unit.

Aspect 32: The optical system of Aspect 1, wherein multiple pixelpatches produce similar or identical overlapping retinal imagesindependent of where the multiple pixel patches are arranged across thesee-through near eye display.

Aspect 33: The optical system of Aspect 1, wherein a patch unit focusesbeams from the patch unit to appear to an eye of the user as though animage or subimage produced by the patch unit is located at an infinitedistance or infinite conjugate, wherein light rays from the patch unitform constant diameter beams traveling away from the patch unit towardsthe eye of the user, and wherein as the beams propagate they divergefrom a center on-axis beam.

Aspect 34: The optical system of Aspect 1, wherein the see-through neareye optical module is attached to, connected to, or embedded within anoptic.

Aspect 35: The optical system of Aspect 34, wherein the optic is aneyeglass lens or a contact lens.

Embodiments of the invention also include a computer readable mediumcomprising one or more computer files containing applications,frameworks, libraries, and such, comprising a set of computer-executableinstructions for performing one or more of the calculations, steps,processes and operations described and/or depicted herein. In exemplaryembodiments, the files may be stored contiguously or non-contiguously onthe computer-readable and/or device-readable medium. Embodiments mayinclude a computer program product comprising the computer files, eitherin the form of the computer-readable medium comprising the computerfiles and, optionally, made available to a consumer through packaging,or alternatively made available to a consumer through electronicdistribution. A s used in the context of this specification, a“computer-readable medium” is a non-transitory computer-readable mediumand includes any kind of computer memory such as floppy disks,conventional hard disks, CD-ROM, Flash ROM, non-volatile ROM,electrically erasable programmable read-only memory (EEPROM), memorycard, and RAM. In exemplary embodiments, the computer readable mediumhas a set of instructions stored thereon which, when executed by aprocessor, cause the processor to perform tasks, based on data stored inthe electronic database on the computer or cloud, or memory describedherein. The processor may implement this process through any of theprocedures discussed in this disclosure or through any equivalentprocedure.

In other embodiments of the invention, files comprising the set ofcomputer-executable instructions may be stored in computer-readablememory on a single computer or distributed across multiple computers, inpersonal communication device and/or devices, or be stored in cloudcomputer. A skilled artisan will further appreciate, in light of thisdisclosure, how the invention can be implemented, in addition tosoftware, using hardware or firmware. As such, as used herein, theoperations of the invention can be implemented in a system comprising acombination of software, hardware, and/or firmware.

Embodiments of this disclosure include one or more computers or devicesloaded with a set of the computer-executable instructions describedherein. The computers or devices may be a general purpose computer, aspecial-purpose computer, personal communication device, personalwearable device, or other programmable data processing apparatus toproduce a particular machine, such that the one or more computers ordevices are instructed and configured to carry out the calculations,sensor data collecting and processing, processes, steps, operations,algorithms, statistical methods, formulas, or computational routines ofthis disclosure. The computer or device performing the specifiedcalculations, processes, steps, operations, algorithms, statisticalmethods, formulas, or computational routines of this disclosure maycomprise at least one processing element such as a central processingunit (e.g., processor or System on Chip (“SOC”)) and a form ofcomputer-readable memory which may include random-access memory (“RAM”)or read-only memory (“ROM”). The computer-executable instructions can beembedded in computer hardware or stored in the computer-readable memorysuch that the computer or device may be directed to perform one or moreof the calculations, steps, processes and operations depicted and/ordescribed herein.

Additional embodiments of this disclosure comprise a computer system forcarrying out the computer-implemented method of this disclosure. Thecomputer system may comprise a processor for executing thecomputer-executable instructions, one or more electronic databasescontaining the data or information described herein, an input/outputinterface or user interface, and a set of instructions (e.g., software)for carrying out the method. The computer (optical) system may includevarious sensors, such as ALS (ambient light sensor) to regulate thebrightness of the display depending on ambient light, or a temperaturesensor that monitors the internal temperature of the TOM to ensure anoperating temperature is within a set, specified, preferred, or “spec”range. The computer system can include a stand-alone computer, such as adesktop computer, a portable computer, such as a tablet, laptop, PDA,wearable device (e.g., electronic watch, smart glasses or HMD—HeadMounted Display), or smartphone, or a set of computers connected througha network including a client-server configuration and one or moredatabase servers. The network may use any suitable network protocol,including IP, UDP, or ICMP, and may be any suitable wired or wirelessnetwork including any local area network, wide area network, Internetnetwork, telecommunications network, Wi-Fi enabled network, or Bluetoothenabled network. In one embodiment, the computer system comprises acentral computer connected to the internet that has thecomputer-executable instructions stored in memory that is operablyconnected to an internal electronic database. The central computer mayperform the computer-implemented method based on input and commandsreceived from remote computers through the internet. The centralcomputer may effectively serve as a server and the remote computers mayserve as client computers such that the server-client relationship isestablished, and the client computers issue queries or receive outputfrom the server over a network.

The input/output user interfaces may include a graphical user interface(GUI), voice command interface, gesture interface, gaze interface, orcombinations thereof, which may be used in conjunction with thecomputer-executable code and electronic databases. The graphical userinterface gesture interface, gaze interface, or combinations thereof,may allow a user to perform these tasks through the use of text fields,check boxes, pull-downs, command buttons, voice commands, variousgestures gaze as a selection mechanism, and the like. A skilled artisanwill appreciate how such user features may be implemented for performingthe tasks of this disclosure. The user interface may optionally beaccessible through a computer connected to the internet. In oneembodiment, the user interface is accessible by typing in an internetaddress through an industry standard web browser and logging into a webpage. The user interface may then be operated through a remote computer(client computer) accessing the web page and transmitting queries orreceiving output from a server through a network connection.

The present invention has been described with reference to particularembodiments having various features. In light of the disclosure providedabove, it will be apparent to those skilled in the art that variousmodifications and variations can be made in the practice of the presentinvention without departing from the scope or spirit of the invention.One skilled in the art will recognize that the disclosed features may beused singularly, in any combination, or omitted based on therequirements and specifications of a given application or design. Whenan embodiment refers to “comprising” certain features, it is to beunderstood that the embodiments can alternatively “consist of” or“consist essentially of” any one or more of the features. Otherembodiments of the invention will be apparent to those skilled in theart from consideration of the specification and practice of theinvention.

It is noted that where a range of values is provided in thisspecification, each value between the upper and lower limits of thatrange is also specifically disclosed. The upper and lower limits ofthese smaller ranges may independently be included or excluded in therange as well. The singular forms “a,” “an,” and “the” include pluralreferents unless the context clearly dictates otherwise. It is intendedthat the specification and examples be considered as exemplary in natureand that variations that do not depart from the essence of the inventionfall within the scope of the invention. Further, all of the referencescited in this disclosure are each individually incorporated by referenceherein in their entireties and as such are intended to provide anefficient way of supplementing the enabling disclosure of this inventionas well as provide background detailing the level of ordinary skill inthe art.

As used herein, the term “about” refers to plus or minus 5 units (e.g.,percentage) of the stated value.

Reference in the specification to “some embodiments”, “an embodiment”,“one embodiment” or “other embodiments” means that a particular feature,structure, or characteristic described in connection with theembodiments is included in at least some embodiments, but notnecessarily all embodiments, of the inventions.

As used herein, the term “substantial” and “substantially” refers towhat is easily recognizable to one of ordinary skill in the art.

It is to be understood that the phraseology and terminology employedherein is not to be construed as limiting and are for descriptivepurpose only.

It is to be understood that while certain of the illustrations andfigure may be close to the right scale, most of the illustrations andfigures are not intended to be of the correct scale.

It is to be understood that the details set forth herein do not construea limitation to an application of the invention.

Furthermore, it is to be understood that the invention can be carriedout or practiced in various ways and that the invention can beimplemented in embodiments other than the ones outlined in thedescription above.

1. An optical system comprising: a see-through near eye optical module comprising a see-through near eye display and a micro-lenslet array, wherein the see-through near eye display comprises one or more plurality of light-emitting pixels and the micro-lenslet array comprises one or more micro-lenslets; wherein one of the one or more plurality of light-emitting pixels is configured as a pixel patch, and wherein the see-through near eye display comprises a plurality of pixel patches; wherein the pixel patch is in optical communication with a micro-lenslet of the one or more micro-lenslets, wherein the pixel patch in optical communication with the micro-lenslet is configured as a patch unit, and wherein the see-through near eye optical module comprises a plurality of patch units; wherein a first image or subimage and a second image or subimage from two or more of the plurality of patch units are directed along a common angular trajectory towards a user wearing the see-through optical module, wherein the first image or subimage is identical to the second image or subimage, wherein a first position of an eye of the user, a head of the user, or both the eye and the head of the user, results in the first image or subimage being directed to a portion of a retina of the user, and wherein when the user changes lateral displacement of the eye of the user, the head of the user, or both the eye and the head of the user, to a second position, the second image or subimage overlaps some or all of the portion of the retina of the user where the first image or subimage was directed; and wherein the see-through near eye display and the micro-lenslet array permit light rays from a physical object in an external environment to pass through the see-through near eye optical module to a retina of the user.
 2. The optical system of claim 1, wherein the see-through near eye display is partially, sparsely, mostly, or fully populated by the plurality of the light-emitting pixels or the plurality of the pixel patches.
 3. The optical system of claim 1, wherein images or subimages from different patch units of the plurality of patch units are at least one of the same, redundant, related, variant, or different, fragments of a larger image.
 4. The optical system of claim 1, wherein two or more of the one or more micro-lenslets have different optical powers or effective focal lengths.
 5. The optical system of claim 1, wherein a first micro-lenslet in a first patch unit has a different optical power or effective focal length than a second micro-lenslet in a second patch unit.
 6. The optical system of claim 1, wherein a first micro-lenslet in a first patch unit has a different optical power or effective focal length than a second micro-lenslet in a second patch unit, and wherein the difference in optical powers or effective focal lengths between the first micro-lenslet and the second micro-lenslet is capable of creating a perception by the user that images or subimages created by the first patch unit and the second patch unit are at different virtual image focal planes, distances, depths, or combinations thereof, from an eye of the user.
 7. The optical system of claim 1, wherein a first micro-lenslet in optical communication with a first group of pixel patches has a different optical power or effective focal length than a second micro-lenslet in optical communication with a second group of pixel patches, and wherein the difference in optical powers or effective focal lengths between the first micro-lenslet and the second micro-lenslet is capable of creating a perception by the user that images or subimages created by the first group of pixel patches and the second group of pixel patches are at different virtual image focal planes, distances, depths, or combinations thereof, from an eye of the user.
 8. The optical system of claim 1, wherein an optical power of a first micro-lenslet of a first patch unit is less than an optical power of a second micro-lenslet of a second patch unit or an effective focal length of the first micro-lenslet of the first patch unit is longer than an effective focal length of the second micro-lenslet of the second patch unit, and wherein beams from the first micro-lenslet appear to emit from a closer virtual image or subimage focal plane, distance, depth, or combination thereof, to an eye of the user.
 9. The optical system of claim 1, wherein micro-lenslet optical power or effective focal length differences across a micro-lenslet array comprising the micro-lenslets are capable of creating a perception by the user that images or subimages are appearing in more than one virtual image focal planes, distances, depths, or combinations thereof.
 10. The optical system of claim 9, wherein a desired virtual image focal plane, distance, depth, or combination thereof, a desired pixel patch or a desired patch unit is illuminated.
 11. The optical system of claim 1, wherein a first micro-lenslet in a first patch unit has a different aperture size than a second micro-lenslet in a second patch unit.
 12. The optical system of claim 1, wherein the optical system includes or supports multiple aperture size micro-lenslets.
 13. The optical system of claim 1, wherein one or more micro-lenslet in a middle area of a user gaze direction have an aperture size larger than one or more micro-lenslet in a peripheral area of the user gaze direction.
 14. The optical system of claim 1, wherein one or more micro-lenslet supporting direct gaze imaging have a larger aperture than one or more micro-lenslet supporting peripheral gaze angles.
 15. The optical system of claim 1, wherein the pixel patches are monochrome pixel patches, and wherein multiple pixel patches are blended together or overlapped to produce a perception by the user of at least one of color, increased resolution, or increased brightness.
 16. The optical system of claim 1, wherein the pixel patches are monochrome pixel patches, and wherein images from the monochrome pixel patches are overlaid on the portion of the user's retina to create a perception by the user of color.
 17. The optical system of claim 1, wherein a subpixel size is offset between a first pixel patch and a second pixel patch or a first patch unit and a second patch unit.
 18. The optical system of claim 1, wherein a center-to-center pitch of a micro-lenslet is offset from an integral number of light-emitting pixels.
 19. The optical system of claim 1, wherein a center-to-center pitch of a micro-lenslet is offset from an integral number of light-emitting pixels, and wherein the offset from the integral number of light-emitting pixels is a fraction of a pixel.
 20. The optical system of claim 1, wherein the optical system produces overlaid images or subimages on the retina of the user, wherein each image or subimage is translated by a fraction of an imaged pixel pitch with respect to one another.
 21. The optical system of claim 1, wherein a first micro-lenslet location is shifted by a fraction of a pixel relative to a second micro-lenslet.
 22. The optical system of claim 1, wherein a first micro-lenslet location is shifted by a fraction of a pixel relative to a second micro-lenslet, and wherein the shift of the first micro-lenslet occurs in both horizontal and vertical dimensions.
 23. The optical system of claim 1, wherein the one or more micro-lenslets are at least one of static micro-lenslets or dynamic micro-lenslets.
 24. The optical system of claim 1, wherein an optical power or effective focal length of the one or more micro-lenslets determines magnification of at least one of the optical system, images, or subimages.
 25. The optical system of claim 1, wherein the optical system comprises multiple pixel patches and/or multiple patch units capable of operating independently of one another.
 26. The optical system of claim 1, wherein the one or more micro-lenslets have a diameter smaller than that of a pupil of an eye of the user.
 27. The optical system of claim 1, wherein the system produces multiple virtual image or subimage focal planes, distances, depths, or combinations thereof, as perceived by the user by overlaying retinal images produced by patch units of the plurality of patch units.
 28. The optical system of claim 1, wherein a primary patch unit is supplemented by one or more additional patch units that are capable of producing an overlaying image or subimage on a retina of the user.
 29. The optical system of claim 1, wherein overlaid images or subimages produced by two or more patch units increase image brightness, increase eye box size, or combinations thereof.
 30. The optical system of claim 1, wherein two or more pixel patches of the plurality of pixel patches produce the same or redundant images on a retina of the user, thereby rendering the optical system capable of locating individual pixel patches of the two or more pixel patches at different areas of the see-through near eye display.
 31. The optical system of claim 1, wherein a first optical power or effective focal length of a primary patch unit differs from a second optical power or effective focal length of a secondary patch unit, wherein the differing optical powers or effective focal lengths of the primary patch unit and the secondary patch unit cause an image or subimage produced by the primary patch unit to appear to be focused at a different virtual image or subimage focal plane, distance, depth, or combination thereof, than an image or subimage produced by the secondary patch unit.
 32. The optical system of claim 1, wherein multiple pixel patches produce similar or identical overlapping retinal images independent of where the multiple pixel patches are arranged across the see-through near eye display.
 33. The optical system of claim 1, wherein a patch unit focuses beams from the patch unit to appear to an eye of the user as though an image or subimage produced by the patch unit is located at an infinite distance or infinite conjugate, wherein light rays from the patch unit form constant diameter beams traveling away from the patch unit towards the eye of the user, and wherein as the beams propagate they diverge from a center on-axis beam.
 34. The optical system of claim 1, wherein the see-through near eye optical module is attached to, connected to, or embedded within an optic.
 35. The optical system of claim 34, wherein the optic is an eyeglass lens or a contact lens. 